Skip to content

Stage 7: land P0 cluster-formation substrate + spec-7.2 IC data-plane#30

Merged
sqlrush merged 63 commits into
mainfrom
stage7-72-integration
Jul 9, 2026
Merged

Stage 7: land P0 cluster-formation substrate + spec-7.2 IC data-plane#30
sqlrush merged 63 commits into
mainfrom
stage7-72-integration

Conversation

@sqlrush

@sqlrush sqlrush commented Jul 9, 2026

Copy link
Copy Markdown
Owner

What this lands

Fuses the Stage 7 P0 cluster-formation substrate with the spec-7.2
inter-connect data-plane decoupling onto main.

P0 lanes (all at tip):

  • xid authority (stage7-p0-xid-authority, tip d8b1d6d414)
  • dead-node diagnosis / async marker (stage7-p0-dead-diagnosis, tip ea2cdc99ee)
  • reconfig liveness (stage7-p0-reconfig-liveness, tip 04ca16191e — folds the
    t/347 join-fence done-epoch axis + t/353 wal_threads_root nightly fixes)
  • formation base (stage7-p0-integration, bfe00bc2e9)

Data plane:

  • spec-7.2 IC data-plane decoupling (D0-D7, incl. F6-1 one-shot conn-reset
    injection + review r2 convergence probe / conn-reset counter / epoch
    short-circuit), tip 38236adf49 fused.

Local verification (macOS, --enable-cluster --enable-cassert)

  • PG 219 core regression: 219/219
  • cluster_regress: 13/13
  • cluster_unit: 160/160

Multi-node kill/recovery TAP legs (t/249, t/347) are not verifiable in the
local macOS sandbox (initdb has no text-search config for the zh_CN.UTF-8
locale → pg_ctl start fails); they are covered by the Linux fast-gate + nightly.

Known-red note

stage6-crossnode-a (t/347 restart leg: Tests: 44 Failed: 0, then pg_ctl
start) is already red on main's own nightly (runs 28934920246 /
28860681493) and is not introduced by this change. No new failing shard is
added relative to current main.

Spec: spec-7.2, spec-4.6a, spec-7.0

SqlRush added 30 commits July 7, 2026 10:11
A locally-mastered GES request enqueued into the work queue had no
wakeup: LMON only drained it when it woke for another reason, worst
case a full heartbeat interval (<= 1s).  Mirror the outbound-ring
enqueue family and SetLatch the drain consumer right after publishing
the slot (publish-before-signal).

Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
…10ms

LAN-scale pacing for the fast-deny / timeout retry path: 10/20/40/80 ms
(total 150 ms) instead of 100/200/400/800 ms (total 1500 ms).  The
per-attempt reply wait is unchanged (cluster.gcs_reply_timeout_ms), so
slow-but-successful serves are unaffected; only the sleep after a fast
denial (DENIED_DEDUP_FULL / DENIED_EPOCH_STALE) or reply timeout
shrinks.  Floor lowered 10 -> 1 ms for tuning headroom; raise the GUC
on slow or congested interconnects.

Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
Both READY publish sites (CR page construct + undo/verdict serve) left
the finished result parked until LMON's next natural wakeup (typically
100-250ms under load, worst case one heartbeat interval).  Kick the
shipper right after the atomic READY store, mirroring the enqueue-side
wakeup family.  Publish-before-signal preserved by the existing
pg_write_barrier + atomic store ordering.

Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
Classify every LMON main-loop duty (24 entries, ClusterLmonDuty) into
never-lazy correctness families (fence / sweeps / clean-leave / node-
remove / reconfig / GRD recovery / BOC / sinval RESET / PI-discard --
order and cadence verbatim, I47-class order writs untouched) and 8
lazy-able queue-consumption families (GES work queue / ship-ready /
grd outbound / sinval out x2 / tt-hint / dedup TTL / backup) gated
behind a producer-set dirty bitmask plus a >= 1 Hz floor.

- producers mark_dirty at enqueue (inside the shared push helpers where
  one exists); dedup TTL is time-based and rides the floor; the backup
  protocol self-re-marks while an operation is in flight (its advance
  steps are fence-driven, not producer-driven)
- cluster.ic_duty_lazy (bool, SIGHUP, default on) is the escape hatch;
  off restores run-every-iteration behavior
- classification is a no-default switch (-Wswitch guards new entries)
  and the unit truth table pins all 24 rows (157/157 unit green)

Baseline: duty chain measured ~140us/iteration avg (max 923us) on a
2-node pair; this removes the fixed per-wakeup dilution tax, it is NOT
the 0.5-1s ship-latency fix (that is the wakeup family + D2 WES).

Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
…kets)

Sixteen log-scale buckets (500us .. 30s + inf) in ClusterGcsBlockShared,
recorded at the single normal-exit funnel of the block request path
(GRANTED / STORAGE_FALLBACK / READ_IMAGE completions; terminal-ereport
exits lose the sample, mirroring the xp scopes).  Always on -- unlike
the GUC-gated xnode-profile scopes -- because this is the ruler for the
ship-latency value gate (p99 < 20ms, p50 < 5ms) and the later wait-
closure acceptance legs.  Exposed as dump category 'gcs' keys
ship_hist_us_le_<bound> / ship_hist_us_inf.

Spec: spec-7.2-ic-data-plane-decoupling.md (D6 slice)
check_GUC_init asserts the C variable initializer matches the GUC
boot_val; the D1 default lowering changed only the registration.
Caught by initdb (cassert) on the first post-change cluster boot.

Spec: spec-7.2-ic-data-plane-decoupling.md (D1)
…rows)

The 16 ship_hist_us_* rows added by the D6 histogram slice land in dump
category 'gcs'; t/110 L1 and t/112 L3 pin the category cardinality.

Spec: spec-7.2-ic-data-plane-decoupling.md (D6 slice)
Optional [node.N] key, host:port, validated like public_addr.  Empty or
absent means the node offers no DATA plane; never derived from
interconnect_addr by offset (adjacent-port collisions on single-host
deployments make an implicit default unsound).  The consumer lands with
the LMS data-plane loop; startup enforcement for multi-node clusters
arrives with the plane flip so intermediate trees keep running without
conf changes.

ClusterNodeInfo grows by the appended field (layout-append rule), which
pushes the 128-slot ClusterConf container past its old 64 KiB anchor --
budget raised to 128 KiB (shmem-only anchor, not a wire contract).

Spec: spec-7.2-ic-data-plane-decoupling.md (D2a)
Introduce ClusterICPlane (CONTROL = 0, DATA = 1) and carve the single
tier1 shmem region into one instance per plane.  Process-local tier1
state (fd arrays, buffers, HELLO machines) is naturally per-process,
so a process's plane is implied: the working alias Tier1Shmem follows
tier1_my_plane (CONTROL by default;  the LMS loop re-aims it via
cluster_ic_tier1_set_my_plane).  Every existing tier1 function -- the
CONNECTED send gate included -- lands on its own plane's peers[] and
listener metadata unchanged;  LMON, backends and SQL observers stay on
the CONTROL instance and contract.

HELLO rides plane + conn_epoch in the documented-zero pad
(capabilities precedent, V1 stays 64B):  plane byte at offset 40,
conn_epoch LE at 44-51, spec-7.3 worker bytes (41/42) reserved zero.
A CONTROL HELLO with epoch 0 stays byte-identical to pre-7.2 (unit
compat pin);  verify paths reject cross-plane HELLOs, and a DATA HELLO
with epoch 0 is refused fail-closed (INV-7.2-CONN-EPOCH substrate).
Plane-selected addresses:  the DATA plane binds/dials data_addr only
-- no data_addr means no DATA plane (LOG + listener off;  never falls
back to the CONTROL address).  RDMA lanes stay CONTROL-owned (DATA is
TCP-only in 7.2).

Spec: spec-7.2-ic-data-plane-decoupling.md (D2b)
The D2b nonzero-ness check false-positived on a fresh cluster: the
initial epoch is legitimately 0 on both sides, so every initial DATA
HELLO was refused in a reconnect storm (caught by the first two-node
smoke).  The sound INV-7.2-CONN-EPOCH receiver check is equality with
this node's current cluster epoch -- a dialer on a stale epoch is
refused and reconnects after observing the bump (fail-closed,
self-healing);  cross-version senders are already excluded by the
HELLO version gate.

Spec: spec-7.2-ic-data-plane-decoupling.md (D2b fix)
New cluster_lms_data_plane.c: the LMS aux process binds the node's
declared data_addr, dials peers by the same mesh role as the CONTROL
plane, runs the HELLO state machines over its own fds, and waits on a
WaitEventSet (DATA sockets + MyLatch) in place of the historic plain
WaitLatch -- the latch-driven park-serve drains are untouched.  No
heartbeats and no liveness scan on this plane: CONTROL (CSSD/LMON) is
the single liveness authority (Q-D2-1);  connect-establishment
timeouts, TCP-error close + backoff reconnect, and requester timeouts
cover the rest.  A node without data_addr runs no DATA plane and LMS
keeps the historic latch-only loop.

No message routes over the plane yet: the GCS block family flips in
the D3/D4 atomic switch;  until then the pump only carries HELLO.

Spec: spec-7.2-ic-data-plane-decoupling.md (D2c)
ClusterPair/Triple/Quad allocate one extra free port per node and
declare it as data_addr in the generated pgrac.conf (allocator-
provided, never offset-derived -- adjacent-port collisions made offset
defaults unsound).  data_ports recorded on the harness object.

Spec: spec-7.2-ic-data-plane-decoupling.md (D2d)
ClusterICMsgTypeInfo gains an owning-plane byte (zero = CONTROL, so
every existing registration is CONTROL without edits) plus the LMS
producer macros (_LMS / _LMS_WORKER / _LMS_DATA; the combined mask
pre-includes the spec-7.3 worker bit).  Two gates enforce the plane:

- dispatch: a frame whose msg_type belongs to the other plane is
  dropped fail-closed and counted (plane_misroute_reject, per-plane
  tier1 instance);  the connection stays up (unlike the peer-level
  unregistered/forged-broadcast rejections).
- physical send (after the dest=self short-circuit, which stays open
  to every mask-allowed logical producer): cross-plane send ereports
  -- an LMS-context emitter of a CONTROL message must stage it into
  the CONTROL outbound ring for LMON (staging rule).

No msg_type is DATA yet: the five GCS block types flip in one commit
at the end of the D3+D4 sequence (no half-migrated window).

Spec: spec-7.2-ic-data-plane-decoupling.md (D3)
…lper

cluster_gcs_send_transition_nowait sent GCS_REQUEST raw; post-flip its
callers (the remote X->S downgrade inside the FORWARD handler and the
BAST-nudge branch) execute in LMS, where a direct CONTROL-plane send
trips the router plane gate by design.  Route through
gcs_send_envelope_or_loopback, whose existing != B_LMON branch stages
the frame into the GRD outbound ring for LMON to send (the D0 staging
case;  fire-and-forget semantics unchanged, reply still HC74-drops).

Spec: spec-7.2-ic-data-plane-decoupling.md (D3)
The dormant half of the D3/D4 atomic flip:

- cluster_lms_outbound.c: DATA twin of the GRD outbound ring (Q6-B;
  single-tail FIFO, LMS is the only consumer, WOULD_BLOCK requeues at
  the HEAD so per-peer byte streams never reorder -- INV-7.2-DATA-FIFO).
  The GCS block REQUEST pre-send hook (direct-land arm) migrates with
  the DATA consumer.  New shmem region + t/020 baselines.
- cluster_lms_wakeup(): SIGUSR1 -> latch (was SIG_IGN in the LMS
  skeleton);  ring enqueue publishes before the kick.
- backend staging entry (enqueue_backend_msg) routes by the msg_type's
  registered plane: DATA -> LMS ring, everything else unchanged.
- cluster_gcs_block_family_on_data_plane(): registry probe (REPLY
  stands in for all five).  The LMON ship_ready / pi_discard tick
  sites skip when the family is DATA;  the LMS loop runs both plus the
  DATA ring drain when it is.  Every branch reads CONTROL today -- the
  flip commit only edits the six registration structs and the whole
  pipeline pivots at once (no half-migrated window).

Spec: spec-7.2-ic-data-plane-decoupling.md (D4)
GetNamedLWLockTranche requires a matching RequestNamedLWLockTranche in
the process_shmem_requests window (I15 pattern);  without it the first
tranche lookup FATALs at startup.  Also stub the register hook for the
shmem unit harness.

Spec: spec-7.2-ic-data-plane-decoupling.md (D4)
Connection-generation fence for the DATA plane (spec-7.2 D5):

- conn_epoch recorded per peer at every CONNECTED transition (dialer
  stamps its HELLO epoch;  accept sides record the peer's), giving the
  connection an epoch identity.
- sender gate in tier1_send_bytes: a DATA frame never goes out on a
  connection bound to a different epoch (HARD_ERROR -> close + re-HELLO
  rebinds at the current epoch).  Cross-epoch interleave on one byte
  stream would defeat the reconfig-rebuild ordering argument (8.A).
  CONTROL is exempt -- its single LMON stream carries the epoch events
  themselves.
- epoch watch in the LMS tick: a bump proactively force-closes the
  whole DATA mesh (reconnect immediately);  the sender gate covers the
  window before the tick notices.
- write-fence linkage: while the fence is enforcing and writes are
  disallowed, LMS sends nothing on the DATA plane (ship_ready /
  pi-discard / ring drain all held;  thaw wakes the latch and the held
  frames drain).  Block images must not leave a fenced node.

Spec: spec-7.2-ic-data-plane-decoupling.md (D5)
Two wait events (F12 four-point symmetry: enum / wait_event.c name
map / views table / count):  ClusterLmsDataRecv is the data-plane
WaitEventSet wait;  ClusterLmsDataSend is the same wait while any peer
holds a backpressured partial frame (waiting for WRITEABLE drainage).
CLUSTER_WAIT_EVENTS_COUNT 118 -> 120;  every count pin across the TAP
+ unit surfaces follows.

Two injection points:  cluster-lms-data-dispatch fires before the
inbound envelope pump of a readable DATA peer (hit surface for the
flip e2e legs);  cluster-lms-conn-reset SKIP forces a one-shot close
of the whole DATA mesh (the reset/epoch legs' trigger) and shares the
epoch-watch reset path.  Registry 158 -> 160 rows (t/015).

Spec: spec-7.2-ic-data-plane-decoupling.md (D6)
…2 D6

The D6 observability commit added two injection points
(cluster-lms-data-dispatch + cluster-lms-conn-reset, registry
158 -> 160) and two wait events (ClusterLmsDataRecv/Send,
118 -> 120) but refreshed only part of the pinned assertion
surface.  Left behind and failing on the branch tip:

- t/015 test 3: the full name-list string_agg lacked the two
  new injection names (the row-count leg alone was updated)
- t/020 L15: total registry size still pinned 158
- cluster_regress cluster_smoke: both wait-event count pins
  (regular + gcluster view) still expected 118

Spec: spec-7.2-ic-data-plane-decoupling.md
An ADG standby without an LMS has no DATA-plane listener, so with
the GCS block family on the LMS-owned DATA plane every cross-node
block request from or to it retransmits to exhaustion.  Mirror the
MRP/RFS standby spawn: the LMS spawn gate accepts PM_HOT_STANDBY
alongside PM_RUN.  Grant ownership stays with LMON (HC4) on every
node.

Spec: spec-7.2-ic-data-plane-decoupling.md
Two-node cold restart deadlocked the DATA mesh: the HELLO verify
layer rejected peers whose declared conn_epoch differed from ours,
but cross-node epochs legitimately differ during the cold-form /
reconfig window, and connection establishment is the very channel
the envelope epoch observation converges through.  Remove the
HELLO-layer conn_epoch rejection (the plane-mismatch rejection
stays) and record the LOCAL current epoch at all three CONNECTED
sites, so the sender-side epoch gate compares a connection against
the same node's view.  Stale frames remain dropped per message by
the envelope HC100 check -- receiver-side stale-epoch protection
lives at the envelope layer, not the HELLO layer.

Spec: spec-7.2-ic-data-plane-decoupling.md
The D5 write-fence linkage paused the WHOLE LMS data-plane send
side while the fence is enforcing and writes are disallowed.  That
turns the write fence into a read fence on the fenced node: the
outbound ring carries only backend solicitations (REQUEST / FORWARD
/ CR + undo fetches -- the master's image REPLY is sent from the
dispatch context and never staged), and those are exactly the
frames a fenced node's own catalog reads and post-crash rejoin
depend on.  A full-cluster cold restart wedged behind its own
reads: ring full, every connection died with a failed-to-enqueue
FATAL, and the starved heartbeats fed a fail-stop reconfig storm.
The ungated CONTROL ring never had this problem pre-flip.

Keep the fence pause on the image-bearing legs (READY serve ship +
PI-discard notes): block images must not leave a fenced node.
Drain the solicitation ring unconditionally.

Spec: spec-7.2-ic-data-plane-decoupling.md
The atomic plane flip (H-5, no half-migrated window): the five
block msg_types (REQUEST / REPLY / FORWARD / INVALIDATE /
INVALIDATE_ACK) register plane = DATA with the producer mask
extended by the LMS family.  REDECLARE alone stays on the CONTROL
plane: recovery re-declare must survive a DATA-mesh teardown
mid-episode, and none of the five migrated handlers emits it.  The
registry probe pivots the LMON tick sites, the LMS loop and the
outbound-ring routing off these six structs, so this one edit moves
dispatch, physical sends and drains onto the LMS-owned tier1
instance.

Riding the flip, as one commit must:

- master-side fail-stop episode fence at the REQUEST wire handler
  entry (phase_for_tag == RECOVERING -> DENIED_RESOURCE_RECOVERING,
  53R9L retry-safe): a stale-view remote requester routes straight
  to the master, so the requester-side acquire gate alone cannot
  close the phantom-holder overtake window
- pgrac.conf post_validate hard gate: multi-node + cluster enabled
  + a real interconnect tier requires data_addr on every node
  (FATAL at startup instead of the first cross-node block request
  hanging)
- pg_cluster_state gcs keys block_family_plane +
  plane_misroute_reject (83 -> 85) with unit stubs; TAP gcs-key /
  wait-event baselines and stale label texts refreshed (t/110-118)
- per-node data_addr for every handwritten multi-node conf in the
  TAP fleet (t/078, 288, 289, 334, 335, 336, 337, 339, 346)
- t/358 data-plane flip e2e: flip fact + DATA listeners on both
  nodes, bidirectional full-table X transfer with wire-request
  growth asserted after the swap (single-block request asserts are
  master-placement dependent: HC72 master==self short-circuits the
  wire REQUEST and rides FORWARD), staging survival via zero plane
  misroutes and zero plane-gate log errors, cross-held read-back;
  online-join write-gate polling and OID burn-align per the t/347
  idiom

Spec: spec-7.2-ic-data-plane-decoupling.md
D6 added 2 injection points (cluster-lms-data-dispatch,
cluster-lms-conn-reset), 2 Cluster: Interconnect wait events
(ClusterLmsDataRecv/Send) and 1 shmem region (lms data outbound), but the
D6 baseline patch only synced t/015 / t/020 / cluster_smoke and missed the
same counts asserted elsewhere (L462 grep-all-assertion-surfaces).

Sync the remaining stale baselines:
  - injection registry 158 -> 160 (+ 316 -> 320 sub-keys)
    t/017, t/018, t/021, t/022, t/023, t/024, t/030
  - Cluster: Interconnect wait events 9 -> 11
    t/010, t/011
  - shmem regions 79 -> 80 (with visibility inject)
    t/021, t/022, t/023

Test-only; no source change. The runtime values were already correct;
only the hardcoded expected counts were stale. All 9 files green.

Spec: spec-7.2-ic-data-plane-decoupling.md
Adds t/360 covering the reachable D7 legs and honest-SKIPping the legs
blocked on a pre-existing substrate wall (user ruling 2026-07-07; see
Stage-7 substrate follow-up register):

  Reachable (real assertions):
    L1  flip fact: block_family_plane=1 both nodes, DATA listeners bound
    L2  value gate: ping-pong X, requester ship histogram p50<5ms, p99<20ms
        (measured p50/p99 <= 500us on loopback, ~40x headroom)
    L3  hygiene: zero plane misroutes, no plane-gate errors

  KNOWN-BLOCKED (SKIP with reason; mirrors t/339's pattern):
    L4  kill -9 LMS x3 -- F1-8 block_device crash-recovery raw_layout_lock
        needs GES, unavailable before the crash-restarting node re-joins
        (recovery-vs-membership substrate gap, orthogonal to the DATA plane)
    L5  conn-reset injection -- F6-1 persistent GUC storm crashes the passive
        LMS with kevent() EBADF; needs one-shot arm + WES fd hardening
    L6  data-dispatch injection -- F6-2 point sits off the actual block recv
        path (hits stay 0); needs repositioning

23 tests, 6 honest SKIPs.  t/348 (3-node) regression green.

Spec: spec-7.2-ic-data-plane-decoupling.md
Publish and seal native-era XID authority after seed shutdown, adopt
prehistory from per-node anchors, and max-merge startup nextXid before
ShmemVariableCache seeding.

Spec: spec-6.15b-xid-authority-native-era.md
A second cluster.enabled=off seed pass on a SEALED authority re-opens the
native era: clear SEALED at bootstrap so a crash of that pass leaves
joiners fail-closed (53RB5, unsealed) instead of silently adopting the
previous pass's stale high-water -- false-invisible for every xid the
crashed pass consumed.  The pass's own clean shutdown re-publishes and
re-seals monotonically.

Adds cluster_xid_authority_begin_native_run(), the bootstrap enabled=off
arm call, unit coverage (seal kept-hw/cleared-flag/idempotent/re-seal),
and t/361 N4 (sealed pass-1 -> unsealing pass-2 boot -> immediate-kill ->
joiner refuses adoption).

Spec: spec-6.15b-xid-authority-native-era.md
SqlRush added 28 commits July 8, 2026 18:24
t/358-361 are reserved by parallel lanes (spec-7.2 / S-xid), t/363 by the
S-dead lane; occupancy verified against origin branches.  The 4-node
formation + kill -9 + convergence run gets its own shard for wall-clock
isolation.

Spec: spec-4.6a-grd-recovery-liveness.md
…adow unit

Same inline suppression + reason as the other cluster_unit main()
signatures (test_cluster_grd.c precedent); a newer local cppcheck
flags it, the CI baseline version does not.
…y baseline + async pre-bump WAIT_EPOCH wedge

The ship-gate nightly on the marker-async head surfaced two lane-introduced
regressions that the fast-gate matrix does not cover:

1. Dump-category baseline miss: reconfig telemetry added a pg_cluster_state
   category (55 -> 56), but three cross-node TAPs that assert the total
   (t/204/205/206) were not updated — only the locally-run L405 subset was.
   Bump them to 56 with the spec-2.29a note.

2. Async pre-bump WAIT_EPOCH wedge (behavior regression): the async fence /
   join-Phase-1 / node-remove marker staging bumps the membership epoch at
   stage-entry but only publishes the reconfig event once the voting-disk
   marker ACKs — now several LMON ticks later instead of the pre-async
   single-tick spin.  In that window the coordinator's OWN GRD recovery IDLE
   tick re-captured the already-bumped epoch as its WAIT_EPOCH baseline, so
   the P0 accept that followed the publish read old == cur and froze the
   affected shards forever — the spec-4.6a section-0 shape, here triggered by
   the coordinator on itself with no IC piggyback (reproduced deterministically
   as t/293 fail-stop remaster + t/326 node-leave shard reopen).

   Fix: while any pre-bump stage is live, the GRD IDLE tick holds its last
   stable pre-reconfig baseline instead of re-capturing
   (cluster_reconfig_has_pending_prebump_stage guard).  This restores the
   pre-async bump->publish atomicity as seen by GRD without touching the epoch
   bump timing or any user-visible reconfig contract.  Adds the
   cluster_grd_recovery_event_old_epoch accessor + a baseline-hold unit test.

Local gates (cassert): unit 158 binaries (incl. baseline-hold test), t/293 +
t/325 + t/326 family all green (were red), t/204/205/206 + observability
TAPs, PG regress 219/219, clang-format/headers/scn-cmp clean.

Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…sion ②b, others-dead)

The spec-2.29a marker-async change turned the clean-leave COMMITTING marker
into an async, multi-tick wait.  In a slow environment (nightly Linux) the
leaving node finishes its drain and stops heart-beating inside that wait
window, so CSSD marks it SUSPECTED->DEAD and bumps the global dead_generation
— and the coherence gate, which compared that SCALAR dead_generation, wrongly
read it as a third-party death intruding mid-drain and escalated an otherwise
healthy clean leave (t/310 dormant-member legs, t/331 clean_leave x idle).
apply_clean_leave_as_coordinator bumps + publishes atomically, so this is NOT
the baseline-hold wedge (②a); it is the coherence predicate counting the
leaving node's OWN expected DEAD.

Fix: compare an others-dead bitmap (the dead set EXCLUDING the leaving node)
instead of the scalar dead_generation, at all four coherence sites
(CL_COHERENT macro, drive_drain commit-point, staged-ACK re-check,
non-staged pre-check).  cl_state snapshots the others-dead set at bind; the
pure predicate does a bitmap compare (fail-closed on a missing view) and is
unit-tested against the reflexive-case matrix.  Every third-party incoherence
still escalates (epoch move OR a non-leaving node entering the others-dead
set); only the leaving node's own expected transition is tolerated.  8.A /
CL-I3 protection is unchanged — argued in spec-2.29a §②b with a case matrix.

Local gates (cassert): policy unit reflexive-case matrix, cluster_unit 158
binaries, PG regress 219/219, clang-format/headers/scn-cmp clean.  NOTE:
t/310/331 are timing-sensitive — the false escalation only reproduces in a
slow environment, so the fix is verified by nightly, not locally (a local
run passes the assertions).  A separate pre-existing cassert teardown SIGABRT
flake in t/310 (isolated: reproduces without this change) is unrelated.

Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…hash)

The P6 all-done gate and the WAIT_EPOCH coordinator witness keyed the
cross-node DONE on event_id, but event_id folds the sender-local
cssd_dead_generation, which drifts with each survivor's private flap
observation history and never converges across nodes: any flap asymmetry
made the survivors compute different ids for the same episode, drop each
other's DONEs, and wedge P6 forever — nondeterministically reintroducing
the permanent-freeze shape this branch exists to remove.

The quorum-accepted dead SET is what actually converges, so the DONE
payload now carries a hash over the dead bitmap alone (same kernel as
the event_id hash, riding the same request-id field pair — no envelope
change), stamped per episode at P0 accept.  Accounting, the P6 gate and
the witness compare against the stamp; event_id stays a purely local
accept-dedup scope.  A late DONE from a previous episode still cannot
back a new one: a coordinator re-election changes the dead set (hash
mismatch) and a same-node re-death rides a higher epoch through the
interposed JOIN bump (epoch conjunct).  Pre-accept frames now mismatch
the previous stamp and are dropped; senders re-announce every tick, so
accounting always lands after the accept snapshot — closing the
first-event pre-accept window as a side effect.

Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R2/R4)
Revert the scope short-circuit that returned NORMAL for a dead static
master's blocks wherever online thread recovery cannot run (the default
configuration and every >2-node deployment): it skipped both the
is_materialized cold-block door and the redo-coverage lost-write door,
so a committed write only the dead node saw could be silently read
stale from shared storage.  No other guard sits on that read path — the
GRD freeze ends with the episode, the HW gate covers only extend
high-water marks, and the page-SCN checks ride the ship path.

Out of scope the posture is now an explicit bounded retryable error
(53R9L, hint updated to name the way out: restart the failed node, or
enable online thread recovery in a supported scope) for every tag whose
static master is dead — including never-written blocks, which the
cold-block door cannot distinguish from not-yet-replayed ones — and the
door reopens the moment the failed node stops being DEAD.  In-scope
behavior is byte-identical.  The now-orphaned scope helper is removed.

t/362 asserts the honest two-outcome contract on fresh connections
(success or explicit SQLSTATE — 53R9L or the TT-unknown visibility
door — never a hang), rides pre-warmed sessions for convergence
observability, and pins the D12 cleanup counter to a positive delta;
t/293's post-kill extend leg follows the same posture.

Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R1/R3)
…grade cleanup

- Order the worker's terminal-result store after its backoff-deadline
  store (pg_memory_barrier) so the relaunch decider never pairs a fresh
  BLOCKED with a stale deadline and skips one backoff wait.
- A structural cause discovered only by the WORKER (SIGHUP race) now
  leaves the retry deadline unset so the FSM's next tick takes the
  MARK_STRUCTURAL branch and emits the once-per-episode operator
  WARNING with the configuration hint.
- Wrap the local S->X upgrade in PG_TRY and release the temporary S
  claim when the ACK wait throws (cancel via CHECK_FOR_INTERRUPTS),
  not only on the false return; document why the completed upgrade
  hard-resets the local S refcount (the X grant subsumes all local S
  declarations).  Linker-only exception stubs for the units that link
  cluster_pcm_lock.o.

Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R5/R6/R7/R9)
…e-bump shmem bit

Two review-r2 items on top of the ②b others-dead coherence fix:

P2-1 — the leaving node's barrier-tick own-commit latch (cl_leaving_barrier_tick)
inferred 'my leave committed' from epoch>baseline AND the others-dead bitmap
unchanged.  The bitmap is not monotone: a third-party node that false-fail-
stopped (bumping the epoch) then recovered leaves CSSD hysteresis DEAD->ALIVE,
so the bitmap rebounds to its bound value while the scalar dead_generation only
advances.  If the leaver's first epoch>baseline observation lands after that
rebound it would mis-latch a leave the survivor coordinator actually refused,
suppress the barrier-deadline escalation, and hang forever in BARRIER_WAIT.
Restore the scalar conjunct (epoch>baseline AND others_dead==bound AND
dead_gen==baseline) via the pure, unit-tested cluster_clean_leave_own_commit_
latched predicate.  This does not reintroduce the ②b false positive: the leaver's
OWN alive->DEAD never bumps ITS OWN dead_generation (a node does not observe
itself dead), so on the leaver side the scalar stays at baseline through its
drain.  Survivor-side coherence keeps the bitmap-only check.  Adds the rebound
negative-leg unit (others_dead unchanged + dead_gen advanced -> escalate).

t/274 — a backend-context coordinator (the fail-stop inject test) advances the
epoch then SYNCHRONOUSLY waits for the fence marker before publishing
(MyBackendType != B_LMON path).  During that seconds-long wait the epoch is
bumped but unpublished, and a concurrent LMON GRD IDLE tick re-captures the
post-bump epoch as its WAIT_EPOCH baseline -> old==cur -> the thread replay
slot never reaches DONE -> HG#1b/HG#5 done-poll timeout.  The ②a baseline-hold
guard only reads LMON-process-local staged flags, invisible to a backend's
pre-bump window.  Publish it in shmem: ReconfigShmem.prebump_sync_active set
before the bump, cleared at every return; the guard reads the shmem bit OR the
local stage.  8.A (durable-marker-before-publish) unchanged.

Reproduced the wedge locally at 87f7930 (HG#1b slot-DONE poll times out
forever), then verified the fix: t/274 goes 18/18 green (exit 2 -> 0).  This
also fixes the long-standing main flake (the wedge is deterministic once LMON
is freed from the sync-marker park; it was merely probabilistic before) — the
t/274 known-flake / L66 registration should be revised.

Local gates (cassert): cluster_unit 158 binaries (incl. own-commit-latched +
version-coherent matrices), t/274 18/18, PG regress 219/219, clang-format/
headers/scn-cmp clean.

Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
The leaving node's barrier-tick own-commit latch must be backed by
direct evidence (the durable COMMITTED marker confirmation for THIS
leave attempt) and must be immune to third-party transient false-DEAD
flap noise: the scalar dead_generation is monotone, so a flap during
the leave window advances it forever and the r2 P2-1 three-conjunct
inference then refuses a healthy committed leave until the barrier
deadline escalates it (nightly t/331 C1/C4 false-escalation, run
28948167577).  The reworked U3b matrix pins:
  (a) evidence, no noise            -> latch
  (d) evidence + flap noise         -> still latch  (RED on current code)
  (c) no evidence, any noise state  -> never latch (deadline escalation
      stays armed; the r2 P2-1 refused-leave mis-latch wedge stays shut)

Known-red TDD commit: leg (d) fails on the current three-conjunct
predicate; the follow-up commit flips the latch to marker evidence.

Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
A crash between write_header_both's two durable renames leaves the
transitioned flag (unseal / CLUSTER_ERA stamp) in the primary copy only,
with a stale pre-transition .bak behind it.  Nothing on the restart path
rewrote the pair -- the bootstrap gates skipped the transition call when
the primary already showed the new flag, and the transition functions
early-returned on the same observation -- so the stale .bak survived the
whole native run.  Any later transient primary read failure then fell
back to it: a stale SEALED .bak hands joiners the previous pass's
high-water (false-invisible + xid reissue), and a stale pre-CLUSTER_ERA
.bak re-opens the native-era re-entry guard.

Make the two flag transitions idempotent re-asserts: the bootstrap calls
them unconditionally on every boot of their respective arm, and they
rewrite BOTH copies until both validate with the post-transition flags
(a missing or invalid copy counts as unsettled and is restored too).
Once both copies are settled the call is a no-write no-op, so the
steady-state boot cost is two header reads.  All fail-closed semantics
(53RB5 refusals, CLUSTER_ERA re-entry FATAL, corrupt-authority PANIC)
are unchanged.

Red-first unit coverage: torn-window tests for both transitions
(complete the transition, re-install the old image as .bak, re-run the
boot path, assert the .bak is repaired and that a primary read failure
falls back to the repaired image, not the stale one).

Spec: spec-6.15b-xid-authority-native-era.md
…osed)

The divergent-lineage prefix check broke out of its comparison loop and
returned CONSISTENT on the FIRST missing local pg_xact page.  That is
only sound for a limit that ends where the local tree ends; pg_xact
front truncation (SimpleLruTruncate removes whole low segments once
every xid they cover is frozen past oldestXid) means a joiner can have
segment 0000 legitimately absent while later segments are present AND
divergent -- the first-page ENOENT then declared the clone CONSISTENT
without ever comparing the surviving pages, and the adopt arm overwrote
the joiner's own live-xid outcome pages with the seed's bits
(false-visible / false-invisible).

Pass the caller's recovery-anchor oldestXid into the check and start
the comparison at oldestXid's own 2-bit slot: bits below it are frozen
truth that CLOG never consults again and whose on-disk survival is an
accident of truncation timing, so they are exempt whether the pages
survive or not (neither truncation nor tuple freezing rewrites
surviving CLOG bytes, so this can never hide live evidence).  Within
the comparable range [oldestXid, min(own_next, native_hw)) a missing
local page is an anomaly -- pg_xact always covers [oldestXid, nextXid)
on a well-formed node -- and now returns UNAVAILABLE (fail-closed
53RB5 at the caller), never CONSISTENT.  Sub-byte boundaries at
oldestXid are masked at 2-bit precision.

Red-first unit coverage: front-truncated divergent clone must DIVERGE
(returned CONSISTENT pre-fix), divergence strictly below oldestXid
stays CONSISTENT (frozen exemption, mid-byte lead mask), oldestXid at
the divergent slot DIVERGES, and a hole inside the comparable range is
UNAVAILABLE (the old truth-table leg expecting CONSISTENT for a missing
segment encoded exactly the blind spot and now expects UNAVAILABLE).

Spec: spec-6.15b-xid-authority-native-era.md
…not inference

Nightly t/331 C1 (clean_leave x idle) / C4 (leave_remove x idle) regressed
at the r2 P2-1 head (run 28948167577; parent green): the leaver's
barrier-tick own-commit latch inferred its commit from
epoch>baseline + others-dead bitmap unchanged + scalar dead_generation
unchanged.  The scalar is monotone, so a third-party transient
SUSPECTED->DEAD->ALIVE flap on the leaver's local CSSD view during the
leave window advances it forever; the latch then refuses a leave the
coordinator ACTUALLY committed and the immediate-escalate arm (or the
barrier deadline) aborts it: phase ends ABORTED_ESCALATE -> IDLE, never
'committed', breaking the C1/C4 drains+commits assertions.  Reverting to
bitmap-only would re-open the r2 P2-1 wedge (rebound mis-latch of a
REFUSED leave suppresses escalation forever), so neither inference is
usable.

Replace inference with direct evidence: latch <=> the durable COMMITTED
marker for THIS leave attempt is confirmed.  The evidence already reaches
the leaver with zero extra IO: the coordinator sends the nonce-bound
LEAVE_COMMITTED only after its qvotec ACKs the COMMITTED marker
majority-durable (there is no runtime voting-disk read path on the
leaver, and none is needed - the LMON tick budget this spec protects is
untouched).  cl_committed_handler now routes through a pure identity
gate (self-addressed + currently leaving + per-attempt nonce + committed
epoch past the bound baseline; fail-closed on any mismatch, so a stale
confirmation - and through it a stale COMMITTED marker from a previous
leave of the same node - can never false-latch) and records the attested
committed epoch E before publishing the evidence flag; the barrier tick
consumes E instead of re-reading its possibly-stale local epoch view.

The latch and the P1-V0.7 exit gate collapse into one step (the evidence
IS the durable-truth-source confirmation).  Escalation semantics are
unchanged: no evidence by the barrier deadline -> the existing
cl_escalate path, which is exactly what bounds a refused leave (it never
gets a COMMITTED marker).  Third-party flaps no longer matter: the
coherence observations are now contract inputs the predicate must ignore
(pinned by the U3b unit matrix, red-first in the parent commit) and feed
only a flap-noise LOG at the latch.  Survivor-side coherence sites
(drive_drain / staged-ACK / pre-check) are untouched.

Unit: U3b matrix flipped green; new U3c pins the evidence identity gate
(match / stale-nonce / misrouted / not-leaving / epoch-not-advanced).
Local gates: t/331 x4 all green (6/6), t/310 24/24, t/363 18/18,
t/274 18/18 (prebump shmem-bit stays green), cluster_unit 158 binaries,
cluster_regress 13/13, PG core 219/219, clang-format-18 +
comment-headers clean.

Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
WAIT_BARRIER / WAIT_CLUSTER carried only epoch-coherence guards, so a
survivor whose CSSD detected a coordinator-folded multi-death late kept
announcing REDECLARE_DONE under its stale dead-bitmap hash: peers that
stamped the full set dropped those frames on the composite key and P6 —
which has no timeout by design — wedged permanently (survivor-behind
detection skew).  Add a mid-episode guard that aborts to IDLE on a newer
local event whose dead-bitmap hash differs: the next tick re-consumes
the event, re-stamps recovery_event_bitmap_hash, re-runs recovery
against the fuller dead set and re-announces DONE under the new
composite key.  Same-set dead_generation drift is absorbed without
episode churn (the event_id keeps its local ABA-scoping role).  The
coordinator-behind direction needs no handling: the coordinator's own
later detection bumps the epoch again and the epoch guards abort.

Correct the stamp-site / accounting / hash-helper comments that
overstated the bitmap as quorum-ratified (each node stamps its OWN
accepted event's bitmap; convergence is eventual via the CSSD deadband
plus this guard), and assert a stamped episode hash is never 0 — 0 is
reserved as unstamped in the mark_peer_done drop test, and a JOIN
episode's all-zero bitmap hashes to a fixed nonzero constant.

Unit: survivor-behind growth leg (H({1}) stamped at the folded epoch,
grown {1,2} event arrives -> abort, re-stamp, full-set remaster, DONE
accounting converges) plus a same-set drift no-churn leg; both go red
with the guard disabled.

Spec: spec-4.6a-grd-recovery-liveness.md (r3-P2-2, r3-P3a)
…arrier

hw_remaster_record_terminal stores the backoff deadline before the
terminal result with a release fence between the stores, but the LMON
relaunch decider loaded result-then-deadline with no read-side pairing:
on a weakly-ordered CPU it could still pair a fresh BLOCKED with a stale
zero deadline and skip one backoff wait.  Insert pg_read_barrier()
between the result and deadline loads (attempts needs no fence — it is
written only by the single-writer LMON FSM) and cross-reference the
pairing at the write site.

Spec: spec-4.6a-grd-recovery-liveness.md (r3-P3b)
The serve-gate unseal (static master no longer CSSD-DEAD -> NORMAL) is
heartbeat liveness, not a direct recovery-completion signal.  Document
the traced proof of why returning NORMAL is nevertheless safe: CSSD —
the only heartbeat sender — is spawned by the phase-4 driver at the
PM_RUN transition, after the startup process completed the node's crash
recovery against shared storage; and even under a stale-ALIVE view a
fetch cannot complete against a still-recovering node, because the
master-side handler default-denies until the node is an in-quorum
MEMBER and only the (equally post-PM_RUN) QVOTEC process can establish
the quorum lease after a restart wiped shmem.  Deny replies surface as
bounded 53R9L, endpoint unavailability as bounded 53R90, and no path
falls back to a silent local storage read.  Replaces the previous
wording that asserted recovery-completion semantics without grounding.

Spec: spec-4.6a-grd-recovery-liveness.md (r3-P2-1 prove-safe)
… leg

The Amendment v1.2 (R1) serve-gate fix had no test lock: the t/362
two-outcome legs stay green under a reverted gate (their success arm
accepts a NORMAL serve of a dead-master page) and no unit links the real
phase decision.  Add L4h: after the kill, with the dead node never
restarted, once the observed survivor's own recovery episode reached
IDLE, a FRESH backend on a cold survivor runs a wide
pg_class/pg_attribute scan that MUST fail with exactly 53R9L — no
success arm, bounded return.  An out-of-scope formation never publishes
the dead node's materialization proof, so every dead-master page stays
RECOVERING while the node is down; the scan spans dozens of catalog
pages hashed ~1/4 to the dead node and cold-reads them past the 16MB
test pool, and it touches no dead-node-written rows (heap INSERTs only,
no DDL), so 53R9L is the only legal outcome.  Verified red-first: with
the gate reverted the suite fails at exactly this leg while every other
leg stays green.

The episode-IDLE wait polls the pre-warmed session with the exact query
shape it ran pre-kill and compares in Perl: a cast or operator added to
the SQL performs fresh syscache lookups that themselves cold-read a
dead-master catalog page and trip 53R9L (the very mechanism under
test).  Also accept the 53R51 write-fence rejection in the pre-existing
two-outcome write legs: a transient lease/epoch fence right after the
reconfig is an explicit, retryable fail-closed rejection inside the
same honest contract.

Spec: spec-4.6a-grd-recovery-liveness.md (r3-P1-1 TAP hard leg)
… P0 substrate

Bring the spec-7.2-ic-data-plane branch (25 commits, D0-D7) on top of
stage7-p0-integration so the D7 fault-injection legs (L4/L5/L6) can be
re-evaluated against the formation/reconfig-liveness/xid-authority P0
substrate (spec-6.15b PR#24 + spec-4.6a PR#26 + spec-2.29a PR#25).

Conflict resolutions (cross-surface grd/lmon/gcs + counter baselines):

  cluster_lmon.c   -- keep BOTH the stage7-p0 iter_started_at profiling
                      timestamp AND the spec-7.2 D1 lazy-duty >=1Hz floor;
                      reuse the single iteration-start timestamp for both
                      (TIER_1 and stub loops).
  cluster_lms.c    -- union of includes (cssd + epoch from P0, gcs_block +
                      write_fence from spec-7.2).
  test_cluster_lmon.c -- keep BOTH new unit tests; UT_PLAN 6->7 (both sides
                      bumped 5->6 independently; the union is 7).
  t/015,017,018,020-024,030 -- injection registry 158 base + spec-2.29a
                      (+1 cluster-qvotec-marker-service-hold) + spec-7.2 D6
                      (+2 cluster-lms-data-dispatch/conn-reset) = 161 (was
                      159 on P0 / 160 on spec-7.2 in isolation).  Name list
                      regenerated from the merged cluster_inject.c registry
                      (ORDER BY name); t/030 M5 sub-keys 161x2 = 322.

Verified: clean build (all targets), cluster_unit 160/160 binaries pass,
zero conflict markers remain.  D7 leg re-evaluation follows in later
commits.
The spec-4.6a Amendment v1.2 (R2) composite convergence key gated both
recovery_done_epoch[] and recovery_done_bitmap_hash[] behind one drop gate
in cluster_grd_recovery_mark_peer_done().  A rejoining node never P0-accepts
the JOIN episode as its own FSM episode (JOIN_COMMITTED is published
coordinator-side only), so its local episode-hash stamp stays 0 forever; the
composite gate then dropped every survivor REDECLARE_DONE, the join fence
(cluster_grd_block_view_rebuilt) never lifted, and every joiner-home block
access fail-closed 53R9L permanently.  Nightly t/347 L4-iii (rejoin with
xid_striping) and t/353 L5 (fresh insert on reborn node0) pinned the wedge.

Split the two accounting axes:

  * done_epoch[] accounts unconditionally (monotonic-max), feeding the
    epoch-only join fence.  Sound because REDECLARE_DONE(epoch=E) is
    broadcast only after grd_block_redeclare_scan_complete(E), so a DONE at
    E proves the sender finished its WAIT_BARRIER re-declare at E -- the
    fence's safety condition, independent of the sender's episode dead set.

  * done_bitmap_hash[] stays composite-gated for the P6 cluster gate and the
    WAIT_EPOCH coordinator witness, which still AND both axes and stay
    fail-closed on the withheld hash half (r3-P2-2 multi-death skew
    protection unchanged).

New unit leg test_recovery_idle_joiner_accounts_done_epoch_for_fence asserts
the fence lifts once survivor DONEs land while the composite half stays
withheld; the stale-bitmap and dead-set-growth legs gain a negative
assertion that the epoch axis alone never fires the composite escape.

Spec: spec-4.6a-grd-recovery-liveness.md
The same S-reconfig nightly that pinned the join-fence wedge also flagged
two stale dump-count baselines that lagged already-shipped counters:

  * t/108 pg_cluster_state pcm category 22 -> 23 (spec-4.6a D12
    dead_cleanup_entries).
  * t/249 grd_recovery dump roster 18 -> 31 keys plus the exact sorted key
    list (spec-4.6 2.4 base + spec-5.16 D5 join direction + spec-4.6a
    liveness/episode surface).

Both stay exact-count / exact-roster assertions; the code already emits the
31-key grd_recovery set (verified against cluster_debug.c).

Spec: spec-4.6a-grd-recovery-liveness.md
…KIP t/360 L5

The cluster-lms-conn-reset injection modelled a single epoch-bump DATA-mesh
reset, but its only arm path is the process-local injection GUC, which stays
armed and re-set skip_pending on every LMS tick -- a reset storm that starved
the mesh and (on the passive side) tripped the accept/close WES churn into a
crash, then hit the F1-8 block_device recovery wall.  That made t/360's L5
leg KNOWN-BLOCKED.

Fix: latch the injected reset so it fires exactly ONCE per LMS process, and
only latch after it has actually torn down a live connection (so an arm-
before-connect race retries instead of wasting the single shot).  The real
epoch-bump reset path (cur_epoch != dp_last_epoch) is unaffected.

t/360 L5 is now a real e2e leg (installcheck, 3x stable):
  L5.1  the injected reset fires once (reset-log count +1)
  L5.2  the mesh reconnects -- both nodes emit a fresh "state CONNECTED"
        after the reset (node0 re-dials, node1 re-accepts)
  L5.3  one-shot: exactly one reset event, no per-tick storm

L5.2 proves reconnect via log evidence rather than a post-reset SQL probe:
by that point the value-gate ping-pong has burned fault_t's rows' xmax into
recycled TT slots, so ANY access (read or write) fail-closes with "cluster
TT status unknown" -- the pre-existing #119 recycled-slot wall, orthogonal to
the DATA plane and not caused by the reset.  Block-shipping capability itself
is quantified by the L2 value gate (394 ships, p99 <= 500us).

L4 (F1-8) and L6 (F6-2) stay honest SKIP; the stage7 P0 substrate was
evaluated and does not unblock either (L4 needs the F1 recovery-vs-membership
fix; L6 needs the data-dispatch injection point repositioned -- observability
polish, block-over-DATA already proven by t/358 + the value gate).

Spec: spec-7.2-ic-data-plane-decoupling.md D7 / F6-1
t/353 L5 crashes node0 and has the reborn node take a FRESH HW space lease,
which requires the survivor to complete HW remaster of the dead node's
shards.  HW remaster reads the dead node's per-thread WAL, so on this
shared_data 2-node pair it is only recoverable when cluster.wal_threads_dir
is configured (cluster_hw_remaster_recoverable).  The pair was created with
shared_data but WITHOUT wal_threads_root, so recoverable() returned false,
the remaster was marked BLOCKED_STRUCTURAL, and the reborn node's fresh-lease
DDL/insert/count fail-closed (L5 subtests 28-31, nightly crossnode-b).

Opt the pair into wal_threads_root, matching every other HW-remaster-
dependent kill test (t/247/248/274/293).  This completes the per-thread WAL
harness config that the RACvsRAC/ClusterQuad harness already carries but this
ClusterPair test was missing.

Spec: spec-4.6a-grd-recovery-liveness.md
…t counter, epoch short-circuit

Code review r2 on the F6-1 one-shot conn-reset commit (three findings):

- Finding 1 (over-claim + convergence): the L5.2 note claimed the recycled
  remote-ITL-slot fail-closed boundary blocks ANY access to fault_t.  It is
  actually per-recycled-slot and does not wall a fresh table's new,
  un-recycled xids.  Corrected the wording (t/360 L5.2 comment) and added
  t/360 L5.4/L5.5: after the reset+reconnect, a FRESH coincidence table
  (fault2_t) is X-swapped cross-node, proving block transfer converges (both
  requesters' block_request_count grow, zero plane misroutes) -- the direct
  convergence proof the reconnect log evidence alone did not give.

- Finding 2 (observability): implement the lms_conn_resets counter -- a shmem
  atomic in ClusterLmsSharedState, bumped once per DATA connection torn down
  by a reset (epoch bump in production, injection in the test), dumped under
  pg_cluster_state category='lms'.  t/360 L5.4 asserts it recorded the reset.
  The cluster-lms-data-dispatch injection point stays a deferred follow-up:
  wiring a counter on its currently-unreachable branch would be a
  permanently-zero (false) observable.

- Finding 3 (nit): restore the epoch-bump short-circuit precedence.
  inject_reset is armed only when there is no epoch bump, so a real bump is
  attributed to the epoch (not the F6-1 one-shot) and does not consume the
  one-shot latch; the injection still fires exactly once on a later tick.
  Test-only path; production is unaffected.

Also complete the D6 wait-event snapshot bump (118 -> 120, +2 LMS DATA-plane
wait events) in the stage3/4/5.5/5/beta acceptance unit tests, which the
earlier bump left stale (stage2 + views + retransmit were already 120).
Added the matching cluster_lms_get_conn_resets stub in test_cluster_debug.

Local gates: cluster_unit 160/160 (0 not-ok), cluster_regress 13/13, PG core
219/219, t/358 flip PASS, t/360 PASS (L5 convergence + L4/L6 honest SKIP,
value gate p99 <= 500us).

Spec: spec-7.2-ic-data-plane-decoupling.md
Advances the reconfig P0 contribution from 0d57fa9 to 04ca161,
bringing in the two nightly-surfaced fixes (t/347 join-fence done-epoch
axis 152b579 + t/353 wal_threads_root 04ca161) plus the
dead-master serve-gate/read-barrier hardening. Integration now carries
all three P0 lanes at their tips (xid d8b1d6d, dead ea2cdc9,
reconfig 04ca161) fused with spec-7.2 IC data-plane.

Spec: spec-4.6a, spec-7.2
The spec-7.2 D3+D4 flip e2e test file was missing the mandatory
'Author: SqlRush <sqlrush@gmail.com>' header (CLAUDE.md rule 11); it
never surfaced on the 7.2 lane because that branch was not pushed to a
main-based fast-gate.  Add IDENTIFICATION + Author + copyright to match
the sibling t/360 banner.

Spec: spec-7.2
@sqlrush sqlrush force-pushed the stage7-72-integration branch from 15f90af to b62331e Compare July 9, 2026 08:44
@sqlrush sqlrush merged commit 089a7c6 into main Jul 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant